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This tool is designed for individuals who want to formatively assess a research- 
practice partnership (RPP) in collaboration with its members. It provides guidance 
on how to set and monitor progress on quarterly RPP goals aligned to five critical 


dimensions that researchers and practitioners must address to build and sustain an 
effective RPP. This tool and its related processes enable RPP evaluators and members 
to monitor the health of their RPP by tracking the extent to which it is meeting its 
goals and demonstrating effectiveness, as articulated by the five dimensions and their 
indicators. Using a formative lens rather than a summative one, the tool encourages 
RPP members to purposefully and honestly reflect on their collaborative work and to 
make necessary adjustments over time. This tool is designed so that even individuals 
with limited experience evaluating or participating in RPPs can use it effectively. 
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INTRODUCTION 


This tool is designed for individuals who want to formatively assess a research-practice part- 
nership (RPP) in collaboration with its members. RPPs are “long-term, mutualistic collabora- 
tions between practitioners and researchers that are intentionally organized to investigate 
problems of practice and solutions for improving district [and state] outcomes” (Coburn 

et al., 2013, p. 2). Education RPPs offer structures and processes for bridging research and 
practice and ultimately driving improvements in K-12 outcomes. 


To date, there is limited literature on how to assess the effectiveness of RPPs. Acommonly 
used framework for doing so is Assessing Research-Practice Partnerships: Five Dimensions of 
Effectiveness (Henrick et al., 2017). That framework consists of five dimensions and related 
indicators “intended to guide the development of more specific protocols and measures that 
could contribute to a body of evidence related to partnership effectiveness” (Henrick et al., 
2017, p. 2). The five dimensions, which researchers and practitioners must address to build 
and sustain an effective RPP, are: 


1. Building trust and cultivating partnership relationships. 

2. Conducting rigorous research to inform action. 

3. Supporting the partner practice organization in achieving its goals. 

4. Producing knowledge that can inform education improvement efforts more broadly. 


5. Building the capacity of participating researchers, practitioners, practice organizations, 
and research organizations to engage in partnership work. 


This tool builds on the Henrick et al. (2017) framework by providing guidance on how to set 
quarterly RPP goals aligned to the five dimensions and related indicators and how to monitor 
progress toward the goals. The tool enables RPP evaluators and members to monitor the 
health of their RPP by tracking the extent to which it is meeting its goals and demonstrating 
effectiveness, as articulated by the five dimensions and related indicators. Monitoring prog- 
ress toward the dimensions can help prevent or address common RPP challenges, such as 
frequent turnover, lack of trust and a common language, and problems caused by complex 
contexts (Coburn & Penuel, 2016). Using a formative lens rather than a summative one, the 
tool encourages RPP members to purposefully and honestly reflect on their collaborative 
work and to make necessary adjustments over time. It is designed so that even individuals 
with limited experience evaluating or participating in RPPs can use it effectively. 


RPP members who piloted this tool in a district-level RPP shared the following feedback: 


| think that the structure that you guys are putting in place here is really 
essential. It’s going to change the way we do business, and | think that’s 
really important. 

—RPP district representative 


It has been a breath of fresh air to have another partner joining us in this 
work, and to actually get us to look up...at the larger context of the project. 
That’s been something that has been fun and informative. 

—RPP research representative 


OVERVIEW OF THE TOOL 


The tool is an Excel workbook that consists of three parts, which are available at https:// 
go.usa.gov/xAV29: 


e Prioritizing dimensions and indicators. The tool’s first worksheet aims to lead researchers 
and practitioners to a consensus on the dimensions and indicators they want to prioritize 
during a predetermined time frame (for example, the first quarter of the school year). 


e Establishing short- and medium-term goals and measures. The tool’s second worksheet 
is designed to support researchers and practitioners in establishing short- and medi- 
um-term goals for each of the RPP’s prioritized indicators. These goals should be measur- 
able and timebound. The short-term goals for the RPP should be able to be accomplished 
during the first year of using this tool, and the medium-term goals should be able to be 
accomplished during the second or third year of using this tool. The long-term goal is the 
same for each indicator: to provide evidence that there are processes, structures, and 
practices ingrained in the partnership that support the given indicator. Therefore, the 
worksheet does not prompt the RPP to develop a long-term goal. 


e Interview protocol. The interview protocol is designed to help the evaluator conduct 
interviews with the RPP members to determine the extent to which the short-term goals 
aligned to the prioritized indicators were achieved, which barriers are preventing prog- 
ress, and which potential structures and process might need to be established or revis- 
ited. The protocol consists of nine worksheets: an overview of the interview protocol for 
the interviewee, a consent form, a worksheet for each of the five dimensions (Dimension 
1 through Dimension 5 tabs), a worksheet for additional reflections (Wrap-Up tab), anda 
worksheet with the rating categories the evaluator can use (Rating Categories tab). Not all 
questions in the interview protocol are intended to be completed; only those tied to the 
indicators prioritized by the RPP each quarter. Each dimension’s worksheet includes six 
key components: 


1. Dimension. Specific features essential to effective RPP design and implementation, 
based on rigorous research. The tool includes the dimensions identified by Henrick 
et al. (2017), which are listed at the top of each dimension worksheet. 


2. Indicator. Discrete characteristics of each dimension that describe where to look for 
evidence of progress on a particular dimension and that suggest the necessary prac- 
tices, systems, and structures that an RPP needs in order to be effective. 


3. Questions. Questions designed to gather information and evidence to determine how 
well an RPP is implementing the necessary practices, systems, and structures that it 
needs in order to demonstrate progress on a particular indicator. 


4. Goals for the Quarter. Space to enter the specific goals established for each prioritized 
indicator in a dimension. 


5. Evidence/Notes. Space to reference evidence for the rationale of a rating, make com- 
ments, and suggest next steps for continuous improvement. 


6. Ratings. The current level of implementation for each prioritized indicator in each 
dimension. 


USING THE TOOL 


This section discusses how to use each part of the tool. 


Guidance for prioritizing dimensions and 
indicators 


The evaluator should plan to facilitate a 60 minute discussion to prioritize the RPP’s dimen- 
sions and indicators. At a minimum, the RPP members from the practice organization who 
should participate include: 


e Champions. Decisionmakers who have the power to commit organizational resources to 
the RPP’s research projects (Kochanek et al. 2015). 


e Facilitators or managers. Individuals who manage the day-to-day work of the RPP’s 
research projects. 


e Primary investigators. Individuals who lead the RPP’s research projects. 


Because RPPs vary in composition, this general guidance should be adapted so that individ- 
uals who play a major role in the RPP’s management and scope of work and who ultimately 
make decisions informed by the results of its research are included in the goal-setting 
process. 


To begin the prioritization process, the evaluator should provide a copy of the Prioritizing 
Dimensions worksheet from the Excel workbook to all the RPP members and ask them to 
read the indicators for Dimension 1. After everyone has read the indicators, the evaluator 
should ask the members to reflect on whether the dimension should be a priority for the 
next quarter. If they agree that it should be, the evaluator should invite members to rank the 
dimension’s indicators according to what they deem most important and feasible to address 
in the next three months. The evaluator should then ask the members to share their rank- 
ings aloud. 


This process provides an opportunity for areas of agreement and disagreement to surface 
and for the RPP to build consensus. If there are discrepancies, the evaluator should invite 
the members to explain their rationale for prioritizing specific indicators. After the members 
have listened to one another, the evaluator should ask whether any members would like to 
modify their initial ranking and if so, to share the new ranking. 


The evaluator should then determine whether there is a consensus. If the members are 
unable to come to a consensus, the evaluator should ask those who rated an indicator as 
most important whether their organization and the RPP have the capacity to make progress 
on it in the upcoming quarter. If the answer is yes, the evaluator should include the indicator 
among the RPP’s priorities; otherwise, the evaluator should ensure that the indicator is con- 
sidered for prioritization in future meetings. With a large RPP the evaluator might establish 


at the outset that the lead representatives of the research and practice organizations will 
make the final decisions regarding the prioritized dimensions and indicators. 


After this process has been completed for the first dimension, it should be repeated for the 
remaining dimensions. 


The capacity to prioritize dimensions will vary across and even within RPPs based on com- 
peting priorities and demands. To increase the likelihood that the RPP makes progress on its 
priorities, the evaluator should encourage RPP members to prioritize only one or two indica- 
tors in a dimension and only three dimensions at a time. 


Some RPPs might not prioritize a particular dimension because of their level of maturation. 
For example, newer RPPs might want to prioritize the dimensions “Building trust and cul- 
tivating partnership relationships” and “Conducting rigorous research to inform action,” 
whereas mature RPPs might have already spent substantial time on those dimensions and 
prefer to focus on other dimensions. 


Guidance for establishing short- and medium- 
term goals and measures 


After the indicators in each prioritized dimension have been ranked, the RPP should set 
measurable, timebound short- and medium-term goals for each prioritized indicator. The 
evaluator should plan to facilitate at least a 30 minute discussion involving, at a minimum, 
the individuals who prioritized the RPP’s dimensions and indicators (that is, the champions, 
facilitators or managers, and primary investigators). 


The RPP might set a short-term goal during the first year in which it is using this tool and 
medium-term goals for the second and third years. The long-term goal is the same for each 
indicator: to provide evidence that there are processes, structures, and practices ingrained 
in the RPP that support the given indicator. Therefore, the worksheet does not prompt the 
RPP to enter a long-term goal in the form. RPPs might aim to achieve this long-term goal for 
each prioritized dimension after the third year of using this tool. 


Even after the long-term goal has been accomplished, challenges will arise that require pro- 
cesses, structures, and practices to adjust. Therefore, prioritizing indicators as well as setting 
and measuring progress toward goals should continue over the RPP’s lifespan. 


Guidance for the interview protocol 


After the RPP has prioritized the dimensions and set goals aligned to the prioritized indica- 
tors, the evaluator should record the goals in the interview protocol workbook, hide the 
worksheets of dimensions that were not prioritized, and hide the rows of indicators that 
were not prioritized under dimensions that were prioritized. Then the evaluator should 
schedule interviews with each RPP member. Each interview should last no more than one 
hour. The evaluator should ask the questions in the order they are listed in the protocol. 
After all the interviews have been completed, the evaluator should assign a rating to each 


prioritized indicator in a dimension. The following category descriptions should inform the 


evaluator’s rating. 


Partnership Health Rating Categories 


Red 


Orange 


Yellow 


Green 


The partnership shows little to no evidence of implementing activities 
aligned to this indicator. 


The partnership shows some evidence of implementing activities 
aligned to this indicator. 


The partnership shows evidence of implementing activities aligned 
to this indicator, but processes, structures, and practices are not well 
ingrained within the partnership. 


The partnership shows evidence of implementing activities aligned 
to this indicator well, and processes, structures, and practices are 
ingrained within the partnership. 


The ratings describe the degree to which RPP members—both researchers and practitioners 
—believe that RPP processes, structures, and practices are in place. These ratings should be 
used to organize the formative feedback for the RPP. Evaluators should highlight where the 
RPP made progress toward its quarterly goals and where there might be room for improve- 
ment. If RPP members do not feel satisfied with the progress made for a given indicator, the 
evaluator should share specific examples of challenges or barriers mentioned during the 
interviews and invite the RPP to consider how to address them in the next quarter. 


TOOL DEVELOPMENT PROCESS 


The interview questions that map to each of the indicators in Henrick et al. (2017) were 
informed by interviews with researchers and practitioners who represented eight RPPs 

at the state and district levels from across the country (Kochanek & Scholz, 2020) and by 
literature on collaborative research processes (such as Kochanek et al., 2015). This tool was 
further modified after testing its relevance, feasibility, and utility with an RPP funded by 
the National Science Foundation. The goal was to ensure that RPPs within and outside the 
Regional Educational Laboratory system can use the tool. 


REFERENCES 


Coburn, C. E., & Penuel, W. R. (2016). Research-practice partnerships in education: Outcomes, 


dynamics, and open questions. Educational Researcher, 45(1), 48-54. 


Coburn, C. E., Penuel, W. R., & Geil, K. E. (2013). Research-practice partnerships: A strategy for lever- 
aging research for educational improvement in school districts. William T. Grant Foundation. 


Henrick, E. C., Cobb, P., Penuel, W. R., Jackson, K., & Clark, T. (2017). Assessing research-practice part- 
nerships: Five dimensions of effectiveness. William T. Grant Foundation. 


Kochanek, J., & Scholz, C. (2020). An exploratory study of how to use research-practice partnerships 
to build trust and support the use of early warning indicators. In L. Wentworth & J. Nagaoka (Eds.), 
Early warning indicators in education: Innovations, uses, and optimal conditions for effectiveness. 
Teachers College Record Yearbook. 


Kochanek, J., Scholz, C., & Garcia, A. (2015). Mapping the collaborative research process. Education 
Policy Analysis Archives, 23(121), 1-27. 


REL 2021-057 
February 2021 


This report was prepared for the Institute of Education Sciences (IES) under Contract 
ED-IES-91990018CO002 by the Regional Educational Laboratory Southwest administered by 
American Institutes for Research. The content of the publication does not necessarily reflect 
the views or policies of IES or the U.S. Department of Education, nor does mention of trade 
names, commercial products, or organizations imply endorsement by the U.S. Government. 


This REL report is in the public domain. While permission to reprint this publication is not 
necessary, it should be cited as: 


Scholz, C., LaTurner, J., & Barkowski, E. (2021). Too! for assessing the health of research-prac- 
tice partnerships (REL 2021-057). U.S. Department of Education, Institute of Education 
Sciences, National Center for Education Evaluation and Regional Assistance, Regional Educa- 
tional Laboratory Southwest. http://ies.ed.gov/ncee/edlabs. 


This resource is available on the Regional Educational Laboratory website at http://ies.ed.gov/ 
ncee/edlabs. 


